The present invention relates to a monitoring technique using an image input device such as a camera, or more in particular to a moving object recognition method for automatically detecting a moving object that has entered the imaging view field in distinction from other various objects not to be monitored in the video signal obtained, and an apparatus for automatically tracking and monitoring a moving object detected using the method.
An object tracking and monitoring apparatus using an image input device such as a camera has been widely used. In recent years, however, a monitoring system has come to be required in which instead of a manned monitoring for detecting and tracking a moving object such as a human being, an automobile or a ship on the sea that has entered the view field of monitoring while watching the image displayed on the monitor, an intruding object can be automatically detected and tracked by an image signal input from an image input device such as a camera and a predetermined information or an alarm is given.
In a method for realizing such a system, the input image obtained from an image input device is compared with a reference background image (i.e. the image not including the object to be detected), and the difference in intensity is determined for each pixel, and an area with a large intensity difference is detected as an object. This method is called a subtraction method and has been widely used.
An application of the object tracking method using the subtraction method described above is the invention disclosed in U.S. Ser. No. 09/078,521 (EP 98108707.5) assigned to the same assignor as the assignee of the present application. The tracking method using the subtraction method poses the problems of a splitting in which a single object is observed and tracked temporarily as a plurality of objects and a coupling in which a plurality of objects are observed and tracked as a single object temporarily. These phenomena occur when detecting an object having a intensity near to that of the background image. In the invention disclosed in U.S. Ser. No. 09/078,521 (EP 98108707.5) described above, the change of connection state of a detected object between successive frames is classified into five states including appearance, single coupling, coupling, disappearance and separation, and these connection state changes are stored sequentially together with information such as the center of gravity or the detection range or the size of the object. Further, the inconveniences of the splitting or coupling are corrected based on the connection state change and information of the detected object in each frame thereby to realize an accurate tracking of an object.
As described above, the subtraction method is for detecting an object based on the difference for each pixel between the input image from an image input device such as a camera and the background image set in advance. In other words, the subtraction method is not selective in objects of selection. Another problem of the subtraction method, therefore, is that moving objects other than the object to be detected, such as trees and ocean waves in the view field (objects not required for detection) are detected undesirably.
More specifically, trembling of plants, fluttering of paper, cloth or the like and the intrusion of small living creatures or fallen leaves in the wind on the land and the motion of waves crushing against the wharf, floating trees on the water surface, the sun entering or leaving the clouds, the instantaneous intensity change of the lighthouse or the vehicle light, are not required to be detected in many cases. The conventional subtraction method poses the problem that an accurate tracking of a moving object is impossible in a scene having such an unrequired moving object.
The object of the present invention is to obviate these problems and to provide a reliable method of recognizing a moving object and a reliable apparatus for tracking and monitoring a moving object in which an object to be detected and other objects are distinguished from each other.
In order to achieve this object, according to one aspect of the invention, there is provided a moving object recognition method for recognizing a specified moving object in sequentially input image signals in distinction from other objects, comprising the steps of sequentially detecting at least one moving object in the sequentially input image signals by a subtraction method, sequentially storing, for each input image signal, in a memory at least one characteristic physical quantity which the moving object has, detecting at least one locus of the moving object based on characteristic physical quantities in the input image signals, storing in a memory characteristic physical quantities of the detected locus in such a manner that the characteristic physical quantities are associated with each other as a series of characteristic physical quantities of the moving object, previously setting a test or criterion of determining whether the moving object belongs to the specified moving object or belongs to objects other than the specified moving object, and determining or classifying from the series of characteristic physical quantities, whether or not said moving object is the specified moving object, based on the determining criterion.
According to an embodiment, the criterion or test described above includes the steps of calculating the time during which the locus of a moving object continues to exist from a series of time and positions at which the moving object is detected as included in the correlated series of characteristic physical quantities (referred to as physical quantities) stored in memory, and determining a moving object with the calculated time longer than a predetermined time as an object to be recognized while determining a moving object with the calculated time not longer than the predetermined time as an object not to be recognized.
Preferably, the locus of the detected moving object is determined using more than two input image signal frames, and the time during which the locus continues to exist is calculated using the number of continuous image signal frames at the time when the locus is detected.
According to another embodiment, the criterion or test described above includes the steps of calculating the variations of the change of the position of the moving object from a series of time and positions at which the moving object is detected as included in the correlated series of the physical quantities stored in memory, and determining a moving object with the calculated position change variation smaller than a predetermined value as an object to be recognized and a moving object with the calculated position change variation not smaller than the predetermined value as an object not to be recognized.
According to still another embodiment, the criterion or test described above includes the steps of calculating the maximum size of a detected moving object from the area thereof included in the correlated series of physical quantities stored in memory, and determining a moving object with the calculated maximum size larger than a predetermined value as an object to be recognized and a moving object with the maximum size not larger than the predetermined value as an object not to be recognized.
According to an embodiment, the area included in the series of physical quantities of each moving object is calculated from a circumscribed polygon about the area occupied by the moving object.
According to another embodiment, the area included in the information series of each moving object is calculated from the area occupied by the moving object.
According to another embodiment, the criterion or test includes the steps of calculating the minimum area of each of the moving objects from the correlated series of physical quantities stored in memory and classifying or determining a moving object with the calculated minimum area smaller than a predetermined area as an object to be recognized and a moving object with the calculated minimum area not smaller than the predetermined value as an object not to be recognized.
According to another embodiment, the criterion or test includes the steps of calculating the maximum area of each of the moving objects from the correlated series of physical quantities stored in memory, calculating the minimum area of each of the moving objects from the correlated series of physical quantities stored in memory, and determining or classifying a moving object with the maximum area larger than a first predetermined value and the minimum value smaller than a second predetermined value as an object to be recognized and determining or classifying other moving objects as objects not be recognized. Preferably, the first predetermined value is an estimated minimum area of the object to be recognized and the second predetermined value is an estimated maximum area of the object to be recognized.
According to another aspect of the invention, there is provided a method of recognizing or determining a specific moving object in sequentially input image signals in distinction from other objects, comprising the steps of sequentially detecting a plurality of moving objects in the sequentially input image signals, sequentially storing characteristic physical quantities of the detected moving objects, detecting one or more loci of the detected moving objects based on the physical quantities, storing a group of characteristic physical quantities included in each locus in association with each other as series of characteristic physical quantities of moving objects, and determining whether each moving object is said specified moving object or not based on the characteristic physical quantities including the detection time, position and area of each moving object changing each moment, wherein the determining step includes at least one of the substeps of calculating a time during which the locus of each moving object continues to exist from the detection time and position included in the series of characteristic physical quantities of the moving object and determining whether the moving object is the specified moving object or not from the time during which the locus continues to exist, calculating a position change variation of each moving object from a detection position included in the series of characteristic physical quantities of each moving object and determining whether the moving object is the specified moving object or not from the calculated variation, calculating a maximum area of each moving object from the area included in the series of characteristic physical quantities of the moving object and determining whether the moving object is the specified moving object or not from the calculated maximum area, and calculating a minimum area of each moving object from the area included in the series of characteristic physical quantities of the moving object and determining whether the moving object is the specified moving object or not from the calculated minimum area.
According to still another aspect of the invention, there is provided a moving object recognition method of sequentially detecting moving objects intruding into an imaging view field by the subtraction method and sequentially tracking and monitoring the moving object with an object tracking and monitoring apparatus, comprising the steps of sequentially storing characteristic physical quantities of a moving object detected by the object tracking and monitoring apparatus, detecting a locus of the detected moving object based on the characteristic physical quantities, storing the characteristic physical quantities as a series of characteristic physical quantities of a moving object, previously setting a test for determining whether the moving object belongs to an object to be tracked and monitored, classifying the moving object into an object to be tracked and monitored and an object not to be tracked or monitored from the stored series of characteristic physical quantities, based on the determining test, and tracking and monitoring as an intruding object the moving object classified as an object to be tracked and monitored.
According to a further aspect of the invention, there is provided an object tracking and monitoring apparatus comprising an image input device, an image memory for storing an image sequentially picked up by the image input device, a program memory for storing a program for operation of the object tracking and monitoring apparatus, a central processing unit for activating the object tracking and monitoring apparatus in accordance with the program, a work memory for analyzing the image stored in the image memory, an alarm unit activated by the central processing unit depending on the result of analysis in the work memory for issuing a signal noticeable by a person or an auxiliary animal, and a monitor unit for displaying a monitored image from the central processing unit issued in response to the analysis in the work memory, wherein the program includes code means for sequentially detecting the moving objects stored in the image memory, code means for sequentially storing characteristic physical quantities of the detected moving objects in the work memory, code means for determining one or more tracks or loci of a moving object based on the characteristic physical quantities of the detected moving object, code means for storing a group of characteristic physical quantities belonging to each locus in association with each other in the work memory as a series of characteristic physical quantitites of a moving object, code means for classifying the moving object into an object to be tracked and monitored and an object not to be tracked and monitored based on the stored series of characteristic physical quantitites, and code means for activating the object tracking and monitoring unit in such a manner as to track the moving object classified as an object to be tracked and monitored as an intruding object.
In order to achieve the object described above, according to still another aspect of the invention, the detected object is classified into a plurality of types based on at least one of the shape and the size of the detected area of the detected object.
Specifically, while monitoring the area including the place where a wave is likely to exist in the imaging view field, waves constituting objects not to be tracked and objects to be tracked are recognized in distinction from each other thereby to reduce the detection error. For this purpose, the geometric feature of the detected area for wave detection is utilized. In the case of the ocean monitoring, for example, the distance to the object is generally sufficient long as compared with the height where a camera is installed, and therefore the camera angle of elevation is small as viewed from the object. As a result, as for a wave observed as a swell of the sea surface changing violently, the wavelength is observed in an apparently large magnitude as compared the amplitude of the wave. In other words, the wave is often detected as a horizontally long object. In view of this feature, the present invention is intended to remove the wave detected area from objects to be tracked, based on at least one of the shape and size of the area detected by object detection, while at the same time removing the wave detected area from the objects to be tracked.
According to one aspect of the invention, there is provided an object tracking method comprising the steps of calculating the intensity difference for each pixel between the image obtained by an imaging device such as a camera and the reference background image where no object to be detected is imaged, determining a pixel having a large difference value as the one where an object exists thereby to detect the area of the pixel, and determining whether a detected area is the one detected by error due to a wave based on the shape information of the detected area so that in the case where the detected area is the one detected by error due to a wave, removing the detected area from the category of objects to be tracked thereby to reduce the erroneous detection of a wave.
According to an embodiment of the invention, there is provided an object tracking method comprising the steps of determining the aspect ratio of a circumscribed rectangle about a detected area and classifying the shape of the area as horizontal long or vertically long, calculating the ratio of which the pixels having an object in a detected area represents of the circumscribed rectangle thereof classified as horizontally long, and removing as a wave a detected area in which the ratio of the pixels having an object therein represents not less than a predetermined value.
According to another embodiment of the invention, there is provided an object tracking method comprising the step of removing a detected area, as determined from the shape information thereof, having a vertical size (height) smaller than the maximum size to be determined as a wave (predetermined height) from the category of objects to be tracked, thereby reducing the chance of erroneously detecting a wave of a height not higher than a predetermined value.
According to still another embodiment, there is provided an object tracking method comprising the steps of calculating the histogram of the number of pixels in vertical direction in a detected area having a vertical size larger than the predetermined height, determining the number of pixels (predetermined length) continued in horizontal direction and larger than a predetermined height from the histogram, and determining as a wave and removing from the category of objects to be tracked the detected areas with the number of continuous pixels smaller in number than the number (predetermined length) to be determined as a wave thereby to reduce the chance of detecting a wave by error.
According to a further embodiment, there is provided an object tracking method comprising the steps of determining the aspect ratio of the circumscribed rectangle about a detected area from the shape information of the detected area and classifying the detected area as a horizontally long area or a vertically long area, calculating the ratio which the pixels having an object therein represents of the circumscribed rectangle area for the detected area with the circumscribed rectangle thereof classified as a horizontally long area, removing from the objects to be tracked the detected area determined to represent not less than a predetermined ratio of the circumscribed rectangle thereof, determining as a wave and removing from the objects to be tracked the detected area determined to represent not more than a predetermined ratio of the circumscribed rectangle thereof and also determined to have a vertical size (height) smaller than the maximum size (predetermined height) to be determined as a wave, calculating a histogram of the number of pixels in vertical direction in a detected area determined to have a vertical size larger than the predetermined height value, determining from the histogram the number of pixels (predetermined length) continued in horizontal direction larger than the maximum size to be determined as a wave, and determining and removing as a wave from the category of objects to be tracked the detected area having a smaller number of continued pixels than the size determined as a wave (predetermined length value), thereby reducing the chance of detecting a wave by error.
In another approach to achieving the object of the invention, the detected area determined as a wave is set as a wave candidate, and the series of the wave candidate determination result for a detected object stored in the tracking process is analyzed retroactively, so that the objects are classified into an object to be detected and a wave, and the wave is removed.
According to a still further aspect of the invention, there is provided an object tracking method comprising the steps of calculating the difference of intensity value for each pixel between the image produced from an imaging device such as a camera and a reference background image in which no object to be detected is imaged, determining an area of the pixels having a large difference as the pixels having an object existing therein and sequentially detecting the area of the pixels, tracking the detected object based on the detection position and detection time of the detected object and the detection position and position change with time of the objects detected in the past, determining whether the detected area is the one detected by error due to a wave based on the shape information of the detected area and determining the detected object as a wave candidate in the case where the detected area is determined to have been detected by error due to a wave, sequentially storing the series of the wave candidate determination result of the detected objects, calculating the ratio of the number of detected objects determined as wave candidates retroactively, and determining as a wave and removing from the objects to be tracked any detected object determined as a wave candidate representing the ratio not less than a predetermined value.
According to still another embodiment, there is provided an object tracking method comprising the steps of determining the aspect ratio of the circumscribed rectangle about a detected area and classifying the detected object as horizontally long or vertically long, calculating the ratio which the pixels having the detected object existing therein represent of the circumscribed rectangle area determined as horizontally long, determining as a wave candidate the detected object representing not less than a predetermined ratio of the circumscribed rectangle thereof, storing the series of the wave candidate determination result of the detected object, calculating the ratio of the number of the detected objects determined as a wave candidate retroactively, and determining as a wave and removing from the objects to be tracked the detected area determined as a wave candidate representing not less than a predetermined ratio, wherein the detected area determined as a wave is removed from the category of objects to be tracked thereby to track an object to be tracked while reducing the wave recognition error.
According to a further embodiment, there is provided an object tracking method comprising the step of determining as a wave candidate a detected object having a vertical size (height) smaller than the maximum size (predetermined height) to be determined as a wave, wherein in the case where the ratio of the number of the detected objects determined as a wave candidate retroactively is not less than a predetermined value, the particular detected object is determined as a wave and removed from the objects to be tracked thereby to reduce the erroneous recognition and the chance of erroneously recognizing a wave.
According a still further embodiment, there is provided an object tracking method comprising the steps of calculating a histogram of the number of pixels existing in vertical direction in a detected object having a vertical size (height) larger than a maximum size (predetermined height value) to be determined as a wave, determining the number of pixels continued in horizontal direction larger than a maximum size (predetermined height) to be determined as a wave from the histogram, and setting as a wave candidate a detected object having the number of pixels continued in horizontal direction smaller than a maximum size (predetermined length to be determined as a wave, wherein in the case where the ratio of the number of detected objects determined as a wave candidate is not retroactively less than a predetermined ratio, the particular detected object is determined as a wave and removed from the category of objects to be tracked, thereby reducing both the erroneous wave recognition and the chance of erroneous wave recognition.
According to a still further embodiment, there is provided an object tracking method comprising the steps of determining the aspect ratio of the circumscribed rectangle about a detected object and classifying the circumscribed rectangle as horizontally long or vertically long, calculating the ratio which the pixels of the detected object with the circumscribed rectangle thereof determined horizontally long represents of the circumscribed rectangle area, determining as a wave candidate the detected object representing a ratio not less than a predetermined value with respect to the circumscribed rectangle thereof, determining as a wave candidate a detected object representing not more than a predetermined ratio of the circumscribed rectangle thereof and having a vertical size (height) smaller than a maximum size (predetermined height) to be determined as a wave, calculating a histogram of the number of pixels existing in vertical direction in a detected object determined to have a vertical height larger than a predetermined height value, determining the number of pixels continued in horizontal direction larger than a predetermined height value from the histogram, and determining as a wave candidate a detected object having the number of continued pixels smaller than an apparent horizontal preset size of an object to be detected, wherein in the case where the number of detected objects determined as a wave candidate retroactively represents not less than a predetermined ratio, the particular detected object is determined as a wave and removed from the category of objects to be tracked, thus reducing both the wave recognition error and the chance of wave recognition error.